OPTIMA Mathematical Programming Society Newsletter 79

Derivative free optimization addresses general nonlinear optimization problems in the cases when obtaining derivative information for the objective and/or the constraint functions is impractical due to computational cost or numerical inaccuracies. Applications of derivative free optimization arise often in engineering design such as circuit tuning, aircraft configuration, water pipe calibration, oil reservoir modeling, etc. Traditional approaches to derivative free optimization until late 1990’s have been based on sampling of the objective function, without any attempt to build models of the function or its derivatives. In the late 90’s model based trust region derivative free methods started to gain popularity, pioneered by Powell and further advanced by Conn, Scheinberg and Toint. These methods build linear or quadratic interpolation model of the objective function and hence can exploit some first and second oder information. In the last several years the general convergence theory for these methods, under reasonable assumptions, was developed by Conn, Scheinberg and Vicente. Moreover, recently Scheinberg and Toint have discovered the “self-correcting” property which helps explain the good performance observed in these methods and have shown convergence under very mild requirements. 1 What is derivative free optimization? Derivative free optimization is a class of nonlinear optimization methods which usually comes to mind when one needs to apply optimization to complex systems. The complexity of those systems manifests itself in the lack of derivative information (exact or approximate) of the functions under consideration. What usually causes the lack of derivative information is the fact that the function values are a result of a black-box simulation process or a physical experiment. The situation is often aggravated by the high cost of the function evaluations and the numerical noise in the resulting values. Thus the use of finite difference derivative approximation is typically prohibitive. The numerous applications of derivative free optimization can be found in engineering design, geological modeling, finance, manufacturing, biomedical applications and many other fields. As the available computational power grows the simulation processes become routine and using optimization of complex systems becomes possible and desirable. Thus the number of applications of derivative free optimization grows continuously, which partially explains the continuing growth of the field itself. Another reason for the growth of the field is the recent development of relatively sophisticated algorithms and theory which address the specific needs of the derivative free problems. Here we will discuss some of the recent developments in the theory of model based derivative free methods. We would like to note that the purpose of this article is to focus on the issue of the maintenance of the geometry of the sample sets in model based derivative free methods. Since this is not a survey the list of references is very limited. 2 The role of geometry When it comes to the derivative free optimization, it is clear that most standard optimization approaches do not apply since they rely on Taylor type models and hence require derivatives. Instead various methods of sampling the objective function have been proposed. The most widely used and well-known of them is the Nelder-Mead algorithm [12], popular for its simplicity and effectiveness, but at the same time notorious for its failure to converge even in simple cases. Roughly speaking, what Nelder-Mead does is the following (for description and analysis of the method see [9] and [23]): the objective function is evaluated at n + 1 affinely independent points and the point with the worst function value is selected. The worst point is then reflected with respect to the hyperplane formed by the remaining n points. Depending on the function value achieved at this new sample point the original simplex may be contracted or new simplex may be expanded or contracted along a certain direction and a possible new sample point may be produced and evaluated. The contraction and reflection steps are designed to find progress along a (hopefully) descent direction. In the process the shape of the simplex changes, often adapting itself to the curvature of the objective function. This observed behavior of Nelder-Mead method is what makes it often so successful in practice. However, it is also the cause of its failure to converge (in theory and in practice) – the simplex may change the shape until it becomes “flat” and the further progress is impossible because the sample points are no longer affinely independent and the sample space may become orthogonal to the gradient direction.